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This  summary  report  presents  recommendations  for  standardizing 
the  software  acquisition  and  development  procedures  of  the 
three  services.  The  report  derives  from  the  work  accomplished 
at  the  Joint  Logistics  Commanders  (JLC)  Software  Workshop  held 
at  the  Naval  Postgraduate  School  in  Monterey,  California,  dur¬ 
ing  the  period  of  April  2  to  5,  1979. 

The  JLC  Software  Workshop  was  sponsored  by  the  Software  Man¬ 
agement  Subgroup  as  chartered  by  the  Joint  Policy  Coordinating 
Group  on  Computer  Resources  Management  (JPCG-CRM).  The  pur¬ 
pose  was  to  examine  the  services'  software  acquisition  guide¬ 
lines,  management  procedures,  and  standardization  efforts  to 
determine  if  there  was  a  basis  for  coordination  and  adoption 
of  joint  service  standards.  It  was  expected  that  such  an 
effort  could  improve  the  efficiency  of  software  acquisition 
through  a  common  approach  to  the  understanding  and  documenta¬ 
tion  of  life-cycle  requirements  for  computer  resources. 

The  workshop  was  composed  of  four  panels : 

Panel  A  -  Software  Acquisition/Development  Standards 

Panel  B  -  Software  Documentation 

Panel  C  -  Standards  for  Software  Quality 

Panel  D  -  Software  Acceptance  Criteria. 

Technical  personnel  from  the  Army,  Navy,  Air  Force,  Marine 
Corps,  other  Federal  agencies,  and  industry  participated  in 
the  workshop.  The  panels  were  chaired  by  industry  represen¬ 
tatives  who  were  responsible  for  planning  and  running  the 
panels,  and  for  follow-up  work  sessions  leading  to  their  final 
reports  (Volume  II,  Proceedings  of  the  Software  Workshop) . 


ii 


The  Software  Management  Subgroup  reviewed  the  individual  panel 
reports  and  adopted  recommendations  that  could  realistically 
be  pursued  within  the  Computer  Software  Management  (CSM) 
charter. 
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SECTION  1 


INTRODUCTION 


1.1  BACKGROUND 

The  complexity  of  military  systems  has  steadily  increased 
because  of  the  growing  demands  of  modern  warfare.  In  general, 
these  demands  have  been  met  through  the  use  of  embedded  com¬ 
puters  which  are  integral  to  the  operation  and  support  of  such 
systems.  It  is  commonly  recognized  that  computer-related 
costs  have  been,  and  will  continue  to  be  the  major  driving 
force  in  the  development  and  deployment  phases  of  system 
acquisition.  The  Department  of  Defense  (DoD)  has  shown 
increasing  concern  in  improved  methods  and  policies  in  what  is 
commonly  called  Embedded  Computer  Systems  (ECS)  acquisition. 

To  address  the  problems  related  to  the  acquisition  and  mainte¬ 
nance  of  embedded  computer  systems,  the  Joint  Logistics  Com¬ 
manders  ( JLC )  established  the  Joint  Policy  Coordinating  Group 
on  Computer  Resource  Management  (JPCG-CRM).  The  JPCG-CRM 
chartered  a  subgroup  on  Computer  Software  Management  (CSM)  to 
serve  as  a  focal  point  for  coordination  of  activities  related 
to  the  acquisition  of  computer  software  used  in  support  of 
defense  systems. 

The  mission  of  the  CSM  Subgroup  is  to  review  policies,  proce¬ 
dures,  regulations,  and  standards  relating  to  computer  soft¬ 
ware,  and  forward  specific  recommendations  to  the  JPCG-CRM  on 
critical  areas  related  to  software  acquisition  management, 
including  software  development,  quality,  testing,  and  post¬ 
development  support.  These  recommendations  should  describe 
and  justify  specific  actions  to  be  taken  by  JLC  or  DoD 
agencies.  Furthermore,  such  recommendations  should  aid  in  the 
improvement  and  standardization  of  the  software  acquisition 
process  within  the  JLC  and  DoD  communities. 
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In  reviewing  current  DoD  policy  and  guidance  in  the  area  of 
software  management,  it  appeared  that  available  information 
was  often  conflicting,  redundant,  or  in  some  cases,  lacking. 

A  software  workshop  was  conducted  to  review  areas  in  which 
shortcomings  were  evident,  and  to  make  appropriate  recommend¬ 
ations  for  improvement  and  standardization  of  the  DoD  software 
acquisition  process.  This  report  incorporates  the  findings  of 
that  workshop  into  recommendations  that  can  be  realistically 
pursued  and  which  offer  maximum  benefit  and  improved  software 
life-cycle  management  procedures  for  the  joint  services. 

1.2  WORKSHOP  ORGANIZATION 

The  software  workshop  was  held  in  Monterey  on  April  2  through 
5,  1979.  The  organization  of  the  workshop  is  shown  in  Figure 
1-1.  The  CSM  Subgroup  limited  the  workshop's  area  of  concern 
to  four  critical  topics,  and  established  panels  to  address 
each  topic: 


•  Software  Acquisition/Development  Standards 

•  Software  Documentation 

•  Standards  for  Software  Quality 

•  Software  Acceptance  Criteria. 

Members  of  each  panel  were  selected  by  the  CSM  Subgroup  from 
the  three  services,  non-DoD  federal  agencies,  and  industry. 
Panel  chairmen  were  chosen  from  industry  in  order  to  avoid 
bias.  Prior  to  the  actual  workshop,  detailed  panel  agendas 
were  written  by  the  chairmen  and  sent  to  all  members,  along 
with  a  required  reading  list.  This  enabled  all  personnel  to 
be  prepared  fully  for  the  proceedings  of  the  workshop.  Each 
chairman  organized  his  panel  in  a  manner  that  would  best 
accomplish  the  particular  panel  objectives  (shown  in  Table  1). 


Daily  meetings  of  all  workshop  members  were  held  to  review 
accomplishments  and  discuss  areas  of  general  concern. 

1.3  FOLLOW-UP  ACTIVITIES 

Upon  completion  of  the  panel  sessions,  each  chairman  presen¬ 
ted  the  preliminary  results  of  his  own  panel  to  the  assembled 
workshop  members.  These  preliminary  results  required  follow¬ 
up  efforts  and  coordination  by  the  panel  members.  The  chair¬ 
men  then  prepared  the  final  panel  reports  which  were  submitted 
to  the  CSM  subgroup  on  June  20,  1979,  for  review  and  consider¬ 
ation.  These  final  reports  are  contained  in  Volume  II,  Pro¬ 
ceedings  of  the  Software  Workshop. 

In  subsequent  meetings,  the  CSM  Subgroup  discussed  and  eval¬ 
uated  the  findings  and  recommendations  of  the  workshop.  The 
outcome  of  these  meetings  is  the  set  of  findings  and  recom¬ 
mendations  contained  in  this  report. 
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SECTION  2  -  REPORT  OF  FINDINGS  AND  RECOMMENDATIONS 


2.1  INTRODUCTION 

The  technical  and  management  areas  affecting  software  acquisi¬ 
tion  and  development  as  reported  by  the  four  workshop  panels 
can  be  consolidated  into  five  basic  areas: 

•  Software  Acquisition  Policy 

•  Software  Acquisition  and  Development  Standards 

•  Software  Documentation  Standards 

•  Software  Quality  Assurance  Standards 

•  Software  Acceptance  criteria. 

A  number  of  problems  were  identified  in  each  area. 

2.2  FINDINGS 

2.2.1  Software  Acquisition  Policy 

There  is  no  general  policy  defining  a  common  software  acqui¬ 
sition  framework  for  the  joint  services.  Each  service  has 
implemented  DoDD  5000.29,  "Management  of  Computer  Resources 
in  Major  Defense  Systems,"  somewhat  independently.  As  a 
result,  differing  policies  exist  among  the  services  produc¬ 
ing  differences  in  emphasis  and  nomenclature  with  varying 
interpretations  and  degrees  of  implementation.  For  example, 
there  are  noted  shortcomings  in  the  area  of  planning  for  post¬ 
development  software  support  (i.e.,  support  during  deploy¬ 
ment).  All  services  appear  deficient  in  the  procurement  of 
the  necessary  support  tools  and  documentation. 

2.2.2  Software  Acquisition  and  Development  Standards 

Within  DoD,  there  are  a  number  of  diverse  regulations  and 
standards  covering  the  various  aspects  of  software  acquisition 


(e.g.,  MIL-STD  483/490,  MIL-STD-1679 } .  Some  of  the  standards 
are  service  unique,  such  as  MIL-STD-1679  (Navy)  and  MIL-STD- 
483  ( USAF ) ,  while  others,  such  as  MIL-S-52779  (Army)  have  been 
adopted  in  practice  by  the  joint  services.  For  those  that  are 
unique,  terminology  and  definitions  are  not  always  common  with 
other  standards.  Partial  and  inadequate  standards  coupled 
with  a  lack  of  standardization  of  Data  Item  Descriptions 
(DIDs),  lead  to  delivered  software  products  which  are  often 
unsatisfactory . 

Contractual  acquisition  of  software  has  been  an  area  of 
embedded  computer  system  development  which  has  not  received 
the  tri-service  attention  it  warrants.  MIL-STD-1679  (Navy), 
"Weapon  System  Software  Development,"  sets  forth  a  number  of 
important  concepts  regarding  acquisition  practices,  but  is 
written  in  such  a  way  that  it  does  not  interface  with  other 
existing  standards  and  data  items. 

1 

The  joint  service  commonality  of  acquisition  and  development 
standards  is  feasible  and  essential.  However,  any  effort  to 
develop  a  consistent  set  of  standards  must  be  accomplished 
within  a  common  policy  framework  as  addressed  in  paragraph 
2.2.1.  Only  then  will  the  various  diverse  standards  covering 
software  acquisition  realize  common  terminology  and 
interpretation. 

2.2.3  Software  Documentation  Standards 

As  with  software  acquisition  and  development,  there  are  a  num¬ 
ber  of  diverse  standards  related  to  software  documentation 
within  the  services.  This  applies  to  documents  developed 
within  the  project  management  office  such  as  software  life- 
cycle  management  plans  as  well  as  contractor-delivered  docu¬ 
ments  referred  to  contractually  as  data  items.  Terminology 
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and  definitions  vary  between  documents  used  by  the  services 
even  though  the  subject  matter  is  identical.  There  is  no 
clear  definition  of  when,  by  whom,  and  for  what  purpose  (e.g., 
verification,  maintenance)  various  documents  are  required. 
Moreover,  there  is  no  defined  methodology  for  determining 
minimum  documentation  requirements  for  a  particular  system. 

The  tendency  is  to  procure  either  a  comprehensive  set  of 
documents  or  none  at  all,  rather  than  specifying  the  minimum 
essential  set. 

It  should  be  noted  that  standardization  of  software  documenta¬ 
tion  must  be  closely  coordinated  with  efforts  in  the  policy 
and  acquisition/development  areas  as  defined  in  paragraphs 
2.2.1  and  2.2.2.  For  example,  certain  acquisition  standards 
include  data  requirements  that  properly  belong  in  DIDs,  while 
other  standards  such  as  MIL-STD-1521,  "Reviews  and  Audits", 
should  identify  documentation  to  be  reviewed  at  each 
milestone . 

2.2.4  Software  Quality  Assurance  Standards 

MIL-S-52779,  "Software  Quality  Assurance  Program  Require¬ 
ments,"  has  been  widely  used  since  1974.  Recently,  it  has 
become  an  official  joint  service  standard.  As  a  Software 
Quality  Assurance  (SQA)  Standard,  MIL-S-52779  specifies  con¬ 
tractor  requirements  for  planning  a  software  QA  program. 

Applications  of  this  standard  have  met  with  varying  degrees  of 
success.  There  have  been  instances  where  Government  acquisi¬ 
tion  managers  have  not  considered  MIL-S-52779  acceptable  due 
to  the  imposition  of  additional  schedule  and  budget  require¬ 
ments.  DoD  plant  representatives  and  DCASR  personnel  have  had 
difficulty  in  evaluating  and  monitoring  contractor  SQA  pro¬ 
grams.  In  addition,  enforcing  compliance  of  such  programs 
with  MIL-S-52779  has  been  difficult. 
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Noted  reasons  for  these  difficulties  are  the  lack  of 
well-defined,  consistent  requirements,  differences  in  SQA 
approaches  by  the  services,  and  unavailability  of  experienced 
personnel.  The  latter  problem  is  aggravated  by  inadequacies 
in  SQA  guidance  and  training.  An  additional  problem  is  that 
a  minimum  set  of  documents  governing  SQA  does  not  exist. 
Besides  MIL-S-52779,  this  minimum  set  should  include: 

a.  A  DID  for  SQA  planning. 

b.  An  SQA  handbook  to  help  in  the  interpretation  of  the 
standard/specif i cat ion . 

c.  An  SQA  guidebook  to  provide  background  in  this 
field. 

2.2.5  Software  Acceptance  Criteria 

The  acceptance  of  software  has  been  the  subject  of  a  great 
deal  of  discussion  and  concern.  Basic  concerns  involve  a  lack 
of  recognized  acceptance  criteria,  a  lack  of  DoD  standardiza¬ 
tion,  and  a  lack  of  historical  data  upon  which  to  base  accep¬ 
tance  criteria  and  procedures. 

Unlike  hardware,  successful  development  of  software  cannot  be 
based  on  passing  a  definitive  test  at  the  end  of  development. 
Experience  has  shown  the  end  item  acceptance  approach  to  soft¬ 
ware  as  disastrous.  Acceptance  based  on  this  approach  pro¬ 
vides  no  warning  of  an  unsatisfactory  product  until  a  point  is 
reached  where  recovery  becomes  exceedingly  expensive  in  terms 
of  cost  and  schedule.  There  is  strong  evidence  to  support  the 
view  that  various  software  acceptance  criteria  must  be  applied 
at  well-defined  points  of  the  development  phase  of  the 


software  life  »cycle.  Furthermore,  these  acceptance  criteria 
must  not  be  allowed  to  be  ignored  for  reasons  of  expediency. 

Within  DoD,  there  is  no  standard  set  of  software  acceptance 
criteria.  Consequently,  the  project  manager  has  no  measure  of 
progress  or  assurance  that  the  software  will  perform  at  an 
acceptable  level.  In  order  to  increase  the  quality  of  soft¬ 
ware  delivered  under  military  contracts,  acceptance  criteria 
must  be  applied  at  meaningful  milestones  in  the  software  life- 
cycle.  Acceptance  criteria  must  be  developed  in  consonance 
with  the  acquisition/development  framework  discussed  in  para¬ 
graphs  2.2.1  and  2.2.2. 

Additionally,  an  effort  must  be  made  to  collect  historical 
data  on  software  development  projects.  The  paucity  of  soft¬ 
ware  error  data  for  joint  service  programs  makes  it  impossible 
to  develop  an  accurate  error  model  to  help  predict  software 
reliability.  Furthermore,  the  establishment  of  a  software 
error  data  base  would  aid  in  the  validation  of  criteria  and 
determination  of  those  points  in  the  development  process  where 
criteria  are  best  applied. 

2.3  RECOMMENDATIONS 


2.3.1  Software  Acquisition  Policy 

Develop  a  general  policy  framework  for  the  joint  services  to 
address  the  entire  software  life  cycle  -  the  common  service 
functional  elements  and  milestones,  including  a  unified  set  of 
terminology  and  definitions,  should  be  specified  as  part  of 
this  effort.  This  policy  framework  should  provide  the  founda¬ 
tion  for  formulating  and  revising  software  acquisition  and 
development  standards,  as  described  in  paragraph  2.3.2,  and 
the  software  DIDs,  (paragraph  2.3.3.)  The  elements  of  the 


software  life  cycle  must  be  described  in  sufficient  detail  to 
allow  common  implementation  procedures  by  the  developers, 
users,  and  maintainers  within  each  service.  Appendix  3  of  the 
panel  report  on  Software  Acquisition/Development  Standard, 
Volume  II,  and  AFR  800-14,  dealing  with  acquisition  and  sup¬ 
port  policies  and  procedures  for  computer  resources  in  sys¬ 
tems,  should  be  used  as  the  basis  for  developing  the  required 
set  of  software  acquisition  policies.  This  set  of  software 
acquisition  policies  must  be  consistent  with  defense  systems 
acquisition  policy  as  described  by  the  5000  series  DoD 
directives. 

The  early  planning  and  specification  of  post-development  soft¬ 
ware  support  tools  and  documentation  requires  emphasis.  The 
development  of  policy  in  this  area  should  specify  software 
support  requirements  as  a  key  element  during  the  validation 
and  demonstration  phase.  The  determination  of  these  require¬ 
ments  later  in  the  program,  when  development  is  well  underway, 
is  too  late. 

2.3.2  Software  Acquisition  and  Development  Standards 

Develop  a  single  unified  set  of  acquisition  and  development 
standards  for  joint  service  application  -  this  effort  should 
emphasize  common  terminology  and  definitions,  and  conform  to 
the  software  acquisition  policy  as  specified  in  paragraph 
‘2.3.1. 


It  is  recognized  that  a  long-term  effort  is  required  to 
develop  a  set  of  acquisition  and  development  standards  for 
tri-service  use.  However,  immediate  benefits  can  be  obtained 
by  adapting  MIL-STD-1521A  and  MIL-STD-1679  as  joint  service 
standards.  Long-term  efforts  include  the  revision  of  stand¬ 
ards  such  as  MIL-STD-48 3  and  MIL-STD-490. 
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It  should  be  noted  that  efforts  in  developing  these  software 
standards  must  be  closely  coupled  with  the  software  document¬ 
ation  standardization  effort  addressed  in  paragraph  2.3.3. 

This  coordination  must  ensure  that  all  documentation  formats 
and  content  descriptions  are  implemented  as  DIDs  which  may  be 
referenced  in  the  software  development  standards  where 
appropriate . 

2.3.3  Software  Documentation  Standards 

Define  and  develop  a  comprehensive  set  of  Data  Item  Descrip¬ 
tions  (DIDs)  for  use  in  software  acquisition  -  to  initiate 
this  effort,  the  requirements  for  documentation  should  be 
reviewed  in  light  of  the  recommendations  made  in  Appendix  3  of 
the  panel  report  on  Software  Documentation,  Volume  II.  This 
review  should  examine  these  documentation  requirements  from 
the  aspect  of  acquisition,  development,  and  support  to 
identify  common  and  unique  requirements  in  each  area.  As  in 
the  acquisition  standards  effort,  common  definitions  and  ter¬ 
minology  must  be  agreed  upon.  After  defining  a  complete  set 
of  documents,  a  complete  review  of  existing  data  items  should 
be  conducted  to  see  what  can  be  salvaged.  Based  on  this 
result,  the  actual  DIDs  should  be  developed  to  provide  a  com¬ 
plete  set  for  joint  service  applications.  These  DIDs  must  be 
coordinated  with  applicable  standards.  Specifically,  consid¬ 
eration  should  be  given  to  immediate  incorporation  of  these 
DIDs  into  the  recommended  joint  service  adaptation  of  MIL- 
STD-1521A  and  MIL-STD-1679. 

A  methodology  for  determining  the  minimum  documentation 
required  for  a  given  acquisition  should  be  developed  -  the 
methodology  should  consider  program  characteristics  such  as 
complexity  of  the  system,  program  cost,  support  requirements, 
operating  environment,  etc.  The  methodology  should  be 
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sufficiently  detailed  to  provide  software  acquisition  managers 
with  a  useful  tool  for  determining  documentation  requirements. 

2.3.4  Software  Quality  Assurance  Standards 

Since  MIL-S-52779  has  been  adopted  as  a  joint  service  stand¬ 
ard,  it  only  remains  to  be  applied  by  the  services.  The  fol¬ 
lowing  documents  should  be  developed  to  support  implementation 
of  MIL-S-52779: 

a.  Using  the  Navy  Data  Item  DI-R-2174  and  Appendix  4  of 
the  panel  report  on  Software  Quality,  Volume  II,  as  a 
basis,  generate  a  DID  for  the  contractor's  SQA  plan. 

b.  Using  Air  Force  Pamphlet  74-2  as;  a  basis,  develop  a 
handbook  for  the  interpretation  of  MIL-S-52779. 

In  addition,  the  following  actions  should  be  completed: 

a.  Define  Government  pr.  ces,  procedures,  method¬ 
ologies,  and  respons.  lities  in  the  area  of  SQA. 
(Such  definitions  may  transcend  present  quality 
assurance  organizational  boundaries. )  Develop  an  SQA 
guidebook  from  these  definitions. 

b.  Establish  training  courses  in  SQA  for  DoD  personnel. 
(Development  of  an  SQA  course  and  guidebook  may  be 
accomplished  concurrently.  ) 

2.3.5  Software  Acceptance  Criteria 

Incorporate  in  the  policy  established  in  paragraph  2.3.1,  the 
requirement  to  use  software  acceptance  criteria  at  critical 
milestones  throughout  the  software  life-cycle  acquisition 
process  -  define  and  develop  a  set  of  acceptance  criteria  for 


use  in  software  acquisition  using  Appendix  7  and  8  of  panel 
report  on  Software  Acceptance  Criteria  as  a  point  of  depar¬ 
ture.  These  acceptance  criteria  must  be  defined  in  terms  of 
the  software  development  life  cycle  applied  to  each  milestone, 
and  must  be  incorporated  into  the  software  acquisition  and 
development  standards  effort  as  described  in  paragraph  2.3.2. 
In  addition,  a  guidebook  should  be  developed  describing  the 
application  of  software  acceptance  criteria. 

Identify  specific  programs  for  the  collection  of  software 
error  data  -  the  collection  of  this  data  will  allow  the  devel¬ 
opment  of  error  models  for  measuring  software  performance.  As 
models  are  developed  and  improved,  more  accurate  standards  for 
performance  and  acceptance  criteria  can  be  established  for 
future  software  acquisition. 

2.4  JPCG-CRM  ACTION 

Because  the  recommendations  presented  in  subsection  2.3  are 
mutually  dependent,  a  plan  of  action  and  milestones  (POA&M), 
which  integrates  their  implementation,  was  developed  for  a 
joint  service  software  standardization  program  (see  Section 
3).  This  POA&M  specifies  the  schedule  necessary  to  achieve 
software  acquisition  standardization  among  the  military 
services.  A  central  activity  or  management  group  should  be 
sponsored  by  the  JPCG-CRM  to  assign,  monitor,  and  review  the 
individual  tasks  specified  in  this  POA&M. 
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SECTION  3 


PLAN  OF  ACTIONS  AND  MILESTONES 


The  following  sections  describe  a  suggested  implementation 
plan  for  recommendations  given  in  subsection  2.3.  For  each  of 
the  five  areas  discussed,  a  subplan  is  given  delineating  task 
descriptions  and  schedules. 

It  must  be  emphasized  that  these  five  major  efforts  are  not 
independent  of  one  another,  and,  thus,  intensive  management  of 
and  close  coordination  among  the  areas  are  necessary. 

Each  subplan  indicates  that  completion  of  its  associated  task 
occurs  upon  submission  of  the  task  products  to  the  JPCG-CRM 
for  approval  and  staffing. 

These  subplans,  in  conjunction  with  the  discussions  in  Section 
2,  should  provide  sufficient  background  and  detail  for  imple¬ 
mentation  of  the  standardization  program. 

A  general  schedule  and  summary  of  tasks  follows. 
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GENERAL  SCHEDULE 
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GUIDEBOOK  SHOULD  BE  ORGANIZATIONALLY  INDEPENDENT 


PRESENTATION  TO  CRM 


Jf  * 


PROJECTS  TO  CRM  FOR  REVIEW  AND  STAFFING 


SECTION  4  -  CONCLUDING  REMARKS 


Pull  implementation  of  the  recommendations  in  this  report  will 
lead  to  better  and  more  uniform  software  acquisition  processes 
in  all  services.  The  improved  acquisition  methods  will  assure 
that  software,  when  delivered,  is  more  understandable  and, 
hence,  easier  to  maintain.  The  standardization  of  regulations 
and  document  formats  will  yield  an  immediate  cost  savings 
through  the  elimination  of  redundant  efforts  across  the  ser¬ 
vices.  Ultimately,  these  actions  will  translate  into  cost  and 
schedule  improvements  as  well  as  more  reliable,  maintainable 
systems . 


In  compliance  with  JLC  direction,  the  policy  definition  will 
be  generated  by  the  CSM,  JPCG-CRM. 


